Replace Tinymath with math.js #4492

gulderov · 2023-07-04T08:54:17Z

Description

Migrate from tinymath to math.js

Tinymath has been deprecated for several years, and as part of the modernization plan (#4306), we need to replace it.

Math.js is an extensive math library for JavaScript and Node.js. It features a flexible expression parser with support for symbolic computation, comes with a large set of built-in functions and constants. It is maintained project with good docs.

The evaluate method can replace tinymath.evaluate with security in mind

Examples :
2 + 3
add(params.count, 3)
[a = params.count * 100, f(x) = sqrt(x), f(a)]

Examples that would NOT work:

parse, derivative and evaluate are disabled as risky:
[ h = parse('x^2 + x'), dh = derivative(h, 'x'), dh.evaluate({ x: params.count }) ]

Issues Resolved

closes #3655
closes #3656
closes #3657

Testing the changes

Open [eCommerce] Revenue Dashboard with sample data => Edit => Create new => New Visualization TSVB => Press + Add Series => Press + Add Metric => Last Aggregation Math => Variable name: count, Expression: params.count + 10 => Check graph

Check List

yarn test:jest
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

…t still supporting most functionality Signed-off-by: Danila Gulderov <gulderov@ya.ru>

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

codecov · 2023-07-04T10:53:22Z

Codecov Report

Merging #4492 (f5f0a82) into main (80fe14d) will increase coverage by 0.00%.
Report is 1 commits behind head on main.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #4492   +/-   ##
=======================================
  Coverage   66.49%   66.50%           
=======================================
  Files        3402     3403    +1     
  Lines       65011    65021   +10     
  Branches    10401    10401           
=======================================
+ Hits        43230    43242   +12     
+ Misses      19212    19211    -1     
+ Partials     2569     2568    -1

Flag	Coverage Δ
Linux_1	`34.81% <100.00%> (+0.01%)`	⬆️
Linux_2	`55.31% <ø> (ø)`
Linux_3	`44.61% <ø> (+<0.01%)`	⬆️
Linux_4	`34.89% <ø> (ø)`
Windows_1	`34.83% <100.00%> (+0.01%)`	⬆️
Windows_2	`55.27% <ø> (ø)`
Windows_3	`44.61% <ø> (ø)`
Windows_4	`34.89% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...eseries/public/application/components/aggs/math.js	`18.18% <ø> (ø)`
...er/lib/vis_data/response_processors/series/math.js	`87.23% <ø> (ø)`
...ib/vis_data/response_processors/series/evaluate.ts	`100.00% <100.00%> (ø)`

... and 1 file with indirect coverage changes

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

abbyhu2000 · 2023-07-26T21:15:14Z

src/plugins/vis_type_timeseries/server/lib/vis_data/response_processors/series/evaluate.ts

Do we disable the following math.js functions due to security issues? How do you know these are the only functions that we need to disable?

These functions are marked as unsafe by the mathjs documentation https://mathjs.org/docs/expressions/security.html

A user could try to inject malicious JavaScript code via the expression parser. The expression parser of mathjs offers a sandboxed environment to execute expressions which should make this impossible. It’s possible though that there are unknown security vulnerabilities, so it’s important to be careful, especially when allowing server side execution of arbitrary expressions.

The expression parser of mathjs parses the input in a controlled way into an expression tree or abstract syntax tree (AST). In a “compile” step, it does as much as possible preprocessing on the static parts of the expression, and creates a fast performing function which can be used to evaluate the expression repeatedly using a dynamically passed scope.

The parser actively prevents access to JavaScripts internal eval and new Function which are the main cause of security attacks. Mathjs versions 4 and newer does not use JavaScript’s eval under the hood.

abbyhu2000 · 2023-07-26T21:15:49Z

src/plugins/vis_type_timeseries/server/lib/vis_data/response_processors/series/evaluate.ts

+    createUnit() {
+      throw new Error(MATH_THROW_MSG);
+    },
+    evaluate() {


Is this evaluate() function different than math.evaluate() from line 9?

It is the same evaluate function. The idea is to disable risky functions on the user side. We can find a list of all risky functions in the mathjs documentation: https://mathjs.org/docs/expressions/security.html.

So it is possible to do: mathjs.evaluate('add(2, 2)').
At the same time, it is impossible to evaluate inside the evaluate function itself: mathjs.evaluate('evaluate("add(2,2)")').

abbyhu2000

Thanks @gulderov, no issues functionality-wise, but some questions regarding the disabled functions.

AMoo-Miki · 2023-07-27T10:52:01Z

Thanks for this PR.

parse, derivative and evaluate are disabled

I labelled this as breaking change; if that is not what you meant and the new library can be used just like the old one, lemme know and I will remove the label.

gulderov · 2023-08-01T10:32:58Z

Thanks @gulderov, no issues functionality-wise, but some questions regarding the disabled functions.

@abbyhu2000 Thank you so much!
I have answered all the questions. Please let me know if I need to provide more information

gulderov · 2023-08-02T08:00:38Z

Thanks for this PR.

parse, derivative and evaluate are disabled

I labelled this as breaking change; if that is not what you meant and the new library can be used just like the old one, lemme know and I will remove the label.

@AMoo-Miki, library change could be marked as breaking for sure
As I can see tinymath is more limited than mathjs and disabled functions are not possible in tinymath

It is not a drop in replacement for users. But in most cases it would work as is or with minimal changes

clamp -> ['x=1', 'minimum=2', 'maximum=3', 'min(max(x, minimum), maximum)'] // 2
first -> ['a=[1,2,3]', 'a[1]'] // 1
last -> ['a=[1,2,3]', 'a[a.size()]'] // 3
unique -> ['setDistinct([1,1,1,2])'] // [1, 2]

abbyhu2000

Thanks for the explanations!

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

ananzh · 2023-09-12T22:41:39Z

cypress fail but not caused by this PR.

gulderov added 2 commits July 4, 2023 11:53

Replace tinymath with math.js

c0e737e

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

Disable functions to make the expression parser less vulnerable whils…

efd1972

…t still supporting most functionality Signed-off-by: Danila Gulderov <gulderov@ya.ru>

gulderov requested review from ananzh, kavilla, seanneumann, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub and BSFishy as code owners July 4, 2023 08:54

github-actions bot added the first-time-contributor label Jul 4, 2023

gulderov added 3 commits July 4, 2023 12:22

more generic link id

7fd7342

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

update changelog

3ce717f

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

snap update

55a1c06

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

gulderov added 6 commits July 4, 2023 14:50

version should remain 3.0.0

2887b18

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

snapshots update

e4d60db

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

tests for math.js disabled functions

a46dc9a

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

Merge branch 'main' into tinymath-to-mathjs

852cf06

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

Merge branch 'main' into tinymath-to-mathjs

aca53d5

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

Merge branch 'main' into tinymath-to-mathjs

46f3313

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

gulderov force-pushed the tinymath-to-mathjs branch from 153c7ac to 46f3313 Compare July 10, 2023 08:19

gulderov added 2 commits July 10, 2023 12:48

rerun tests

c6a59b0

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

rerun tests

6fd0370

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

Merge branch 'main' into tinymath-to-mathjs

e50971d

Signed-off-by: Danila Gulderov <gulderov@ya.ru>

abbyhu2000 self-assigned this Jul 26, 2023

abbyhu2000 reviewed Jul 26, 2023

View reviewed changes

AMoo-Miki added the breaking change label Jul 27, 2023

joshuarrrr added the v3.0.0 label Aug 4, 2023

abbyhu2000 previously approved these changes Aug 18, 2023

View reviewed changes

Merge branch 'main' into tinymath-to-mathjs

f5f0a82

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

ananzh dismissed abbyhu2000’s stale review via f5f0a82 September 12, 2023 21:41

ananzh requested a review from curq as a code owner September 12, 2023 21:41

ananzh approved these changes Sep 12, 2023

View reviewed changes

abbyhu2000 approved these changes Sep 12, 2023

View reviewed changes

ananzh merged commit 620624c into opensearch-project:main Sep 12, 2023
53 of 55 checks passed

joshuarrrr mentioned this pull request Oct 2, 2023

CVE-2020-11023 (Medium) detected in jquery-1.11.1.js #4732

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace Tinymath with math.js #4492

Replace Tinymath with math.js #4492

gulderov commented Jul 4, 2023 •

edited

Loading

codecov bot commented Jul 4, 2023 •

edited

Loading

abbyhu2000 Jul 26, 2023 •

edited

Loading

gulderov Aug 1, 2023

abbyhu2000 Jul 26, 2023

gulderov Aug 1, 2023

abbyhu2000 left a comment •

edited

Loading

AMoo-Miki commented Jul 27, 2023

gulderov commented Aug 1, 2023 •

edited

Loading

gulderov commented Aug 2, 2023

abbyhu2000 left a comment

ananzh commented Sep 12, 2023

Replace Tinymath with math.js #4492

Replace Tinymath with math.js #4492

Conversation

gulderov commented Jul 4, 2023 • edited Loading

Description

Issues Resolved

Testing the changes

Check List

codecov bot commented Jul 4, 2023 • edited Loading

Codecov Report

abbyhu2000 Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

gulderov Aug 1, 2023

Choose a reason for hiding this comment

abbyhu2000 Jul 26, 2023

Choose a reason for hiding this comment

gulderov Aug 1, 2023

Choose a reason for hiding this comment

abbyhu2000 left a comment • edited Loading

Choose a reason for hiding this comment

AMoo-Miki commented Jul 27, 2023

gulderov commented Aug 1, 2023 • edited Loading

gulderov commented Aug 2, 2023

abbyhu2000 left a comment

Choose a reason for hiding this comment

ananzh commented Sep 12, 2023

gulderov commented Jul 4, 2023 •

edited

Loading

codecov bot commented Jul 4, 2023 •

edited

Loading

abbyhu2000 Jul 26, 2023 •

edited

Loading

abbyhu2000 left a comment •

edited

Loading

gulderov commented Aug 1, 2023 •

edited

Loading